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PREFACE 


The  Human  Resources  Directorate  of  the  Armstrong  Laboratory,  formerly  the  Air  Force 
Human  Resources  Laboratory,  developed  a  Job  Performance  Measurement  System  to  measure 
the  job  performance  of  first-term  enlisted  personnel.  The  measures  were  desired  and 
developed,  and  data  were  collected  on  the  job  performance  of  first-term  airmen  for  eight  Air 
Force  Specialties  between  1984  and  1987. 

This  report  documents  the  results  of  a  study  conducted  to  optimize  overall  job  performance  of 
personnel  based  on  their  aptitude  test  scores.  This  work  was  performed  by  the  authors  under  in- 
house  Work  Unit  No.  1121-12-00. 

The  authors  are  grateful  to  the  many  government  scientists,  contract  researchers,  subject 
matter  experts  and  base  personnel  for  their  support  in  the  design,  development  and  data  collection 
for  this  project.  A  previous  version  of  this  paper  was  presented  at  the  annual  meeting  of  the 
American  Psychological  Association,  August,  1992. 
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differential  assignment  potential  in  the  ASVAB; 

A  simulation  of  job  performance  gains 

SUMMARY 

This  research  simulated  the  effects  of  using  aptitude  test  scores  to  optimize  overall  job 
performance  of  first-term  enlisted  personnel.  Optimal  reassignment  resulted  in  performance  gains 
above  both  random  assignment  and  current  assignment  allocations.  Recommendations  are  made 
for  the  more  sophisticated  use  of  aptitude  data  and  improved  human  resource  planning. 

L  INTRODUCTION 

The  use  of  tests  for  military  personnel  selection  has  a  well-documented  history  in  the  applied 
literature  (Weeks,  Mullins,  &  Vitola,  1975;  Welch,  Kucinkas,  &  Curran,  1990).  Relatively  little 
attention  however  has  been  devoted  to  the  process  of  classification — allocating  applicants  to  two 
or  more  jobs  based  on  differences  in  the  utility  of  alternative  assignments  (Zedeck  &  Casio, 

1984).  Recent  studies  relating  the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  to 
military  performance  criteria  (Hunter,  1985;  Johnson  &  Ziedner,  1990;  Johnson,  Ziedner,  & 
Scholerios,  1990;  Ree  &  Earles,  1991,  Ree,  Earles,  &  Teachout,  1992, 1994;  Schmidt,  Hunter,  & 
Larson,  1988)  have  shown  equivocal  results.  Hunter  (1985),  Schmidt  et  al.  (1988),  Ree  and 
Earles  (1991)  and  Ree  et  al.  (1992;  1994)  found  that  specific  abilities  added  marginally  to  a 
general  ability  factor  in  predicting  technical  training  and  job  performance  criteria  but  did  not 
document  any  practical  benefits  associated  with  the  gain.  Johnson  and  Ziedner  (1990)  and 
Johnson  et  al.  (1990)  document  that  non-cognitive  measures  (interests  and  psychomotor  tests) 
could  add  to  the  classification  utility  of  the  ASVAB,  if  the  measures  were  selected  to  enhance  the 
differential  content  of  the  test.  Still  at  issue,  however,  is  whether  the  ASVAB  in  its  present  form 
can  be  configured  to  provide  differential  classification  value.  The  purpose  of  the  present  study 
was  to  demonstrate  the  potential  classification  utility  of  the  ASVAB  compared  to  random  and 
current  assignment  practices  and  to  express  the  predicted  performance  g^s,  if  any,  in  the 
equivalent  experience  levels  required  to  obtain  them. 

n.  METHOD 


Subjects 

First-term  enlisted  personnel  in  eight  Air  Forces  specialties  (N  =  1250)  were  followed  firom 
entry  into  service  into  their  first  job  assignments.  The  specialties  included  jet  engine  mechanic, 
aerospace  ground  equipment  mechanic,  information  systems  radio  operator,  personnel  specialist, 
air  traffic  control  operator,  avionics  cominunications  specialist,  aircrew  life  support  specialist  and 
precision  measurement  equipment  laboratory  specialist.  Ethnic  and  demographic  composition  of 
the  group  was  representative  of  all  Air  Force  accessions  during  this  period.  Males  constituted 
83%  of  the  group  and  ethnic  mix  was  78%  white  and  22%  black  or  other.  Average  age  of 
incumbents  was  22  years  and  each  had  spent  an  average  of  28  months  in  service. 
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Performance  Measures 


The  job  performance  of  each  incumbent  was  measured  by  an  in-depth  work-sample  test 
designed  to  assess  maximum  performance  potential  under  ideal  conditions  (for  a  more  complete 
description  of  these  measures  see  Hedge  &  Teachout  (1986, 1992)).  Each  test  contained  detailed 
step-by-step  checklists  that  specified  the  conditions,  standards,  and  behaviors  for  successfiil 
performance  on  a  series  of  tasks  representative  of  the  job  of  the  first-term  enlistee.  Tasks  were 
performed  at  each  individual’s  work  site  under  the  observation  of  a  trained  test  administrator  who 
scored  each  step  on  a  correctAmcorrect  basis.  Incumbents  were  instructed  to  perform  each  task 
according  to  technical  order  (TO)  procedures.  Examinees  were  individually  administered  the 
work-sample  tests  consisting  of  20  to  30  tasks  for  each  specialty.  Administration  of  the  tests 
required  approximately  4  to  7  hours  with  a  maximum  time  limit  specified  for  each  task.  Summary 
scores  were  obtained  from  the  work-sample  test  and  converted  within  each  specialty  to  a  standard 
score  metric  (Mean  =  50;  SD  =  10). 


Aptitude  Tests 


Prior  to  enlistment  in  the  Air  Force,  each  job  incumbent  was  administered  the  Armed  Services 
Vocational  Aptitude  Battery  (ASVAB)  as  part  of  the  entry-level  screening  program  (U.S. 
Department  of  Defense,  1984).  The  battery  yields  10  subtest  scores  measuring  math  and  verbal 
skills  and  technical  knowledge  (i.e.,  auto  and  shop).  ASVAB  scores  are  reported  in  standard 
score  metrics  with  a  mean  of  50,  and  standard  deviation  of  10  and  are  based  on  a  nationwide 
sample  of  American  youth  (U.S.  Department  of  Defense,  1982).  Descriptive  data  on  the  sample 
are  shovra  in  Table  1 . 


Table  1.  ASVAB  Subtest  Means  and  Standard  Deviations  By  Specialty  Ql  =  1250) 


SPECIALTY 


Predictor  JET  AGE  RADIO  PERS  ATC  LIFE  COMM  PMEL 


GS 

53.3  (6.7) 

54.2  (5.8) 

50.8(8.1) 

50.2(7.0) 

55.7(6.3) 

52.4(7.2) 

59.7(5.4) 

59.0(5.1) 

AR 

54.3  (7.0) 

53.8  (6.2) 

52.7(6.5) 

52.9(6.2) 

58.8(5.0) 

53.9(6.2) 

60.7(4.8) 

60.4(4.4) 

WK 

51.6  (5.7) 

52.8  (4.7) 

51.9(5.6) 

52.4(5.6) 

55.2(4.4) 

52.6(5.3) 

56.3  (4.7) 

56.0  (4.S) 

PC 

52.3  (5.8) 

53.2  (6.4) 

53.9(5.6) 

53.6  (5.9) 

56.2(4.6) 

53.2(5.5) 

57.2  (4.7) 

57.2(4.3) 

NO 

51.0(7,2) 

51.9(6.5) 

57.9(4.4) 

57.8(4.3) 

55.6(5.7) 

53.8(6.3) 

55.6(5.6) 

56.2(6.3) 

CS 

50.1  (6.4) 

51.0(6.6) 

56.6(6.3) 

57.7(6.1) 

54.7(6.5) 

53.2(7.0) 

54.3(7,1) 

55.8  (7.6) 

AS 

60.5  (6.7) 

59.4(5.9) 

49.1  (8.5) 

47.7(8.1) 

56.5  (8.0) 

52.4(8.7) 

60.9(5.8) 

59.7(6.8) 

MK 

51.6  (7.7) 

52.8(7,1) 

52.9(7.7) 

53.6  (7.2) 

57.1  (6.8) 

53.2(7.5) 

61.1  (4.6) 

60.7(5.7) 

MC 

57.3  (6.9) 

57.3  (6.2) 

50.4(8.4) 

49.7(7.6) 

57.6(7.3) 

53.2(8.0) 

61.8  (5.5) 

61.5  (5.9) 

El 

56.1  (7.3) 

55.4(7.0) 

49.4(8.3) 

48.1  (7.7) 

55.0(7.6) 

52.0(7.5) 

61.3  (6.2) 

60.7(5.9) 

EXPER 

29.5(11.2) 

28.1  (10.4) 

23.8(13,0) 

27.9(11.8) 

26.9(8.8) 

28.7(11.1) 

35.3  14.9) 

27.4(10.7) 

NOTE.  ASVAB  subtests  are  abbreviated:  GS  =  General  Science;  AR  =  Arithmetic  Reasoning; 
WK  =  Word  Knowledge;  PC  =  Paragraph  Comprehension;  NO  =  Numerical  Operations;  CS  = 
Coding  Speed;  AS  =  Auto  and  Shop  Information;  MK  =  Math  Knowledge;  MC  =  Mechanical 
Comprehension;  El  =  Electronic  Information.  Specialties  are  abbreviated:  JET  =  Jet  Engine 
Mechanic;  AGE  =  Aerospace  Ground  Equipment  Mechanic;  RADIO  =  Information  Systems 
Radio  Operator;  PERS  =  Personnel  Specialist;  ATC  =  Air  Traffic  Control  Operator;  LIFE  = 
Aircrew  Life  Support  Specialist;  COMM  =  Avionic  Communications  Specialist;  PMEL  = 
Precision  Measurement  Equipment  Lab  Specialist.  EXPER  =  Experience. 
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Experience 


Experience  measures  were  recorded  as  months  of  service  between  date  of  entry  into  service 
and  the  time  at  which  the  performance  tests  were  administered. 

Analyses 

Hands-on  work  sample  performance  measures  were  regressed  on  the  ASVAB  subtests  and  the 
experience  measure  separately  for  each  of  the  eight  specialties.  The  least-squares  regression 
equations  were  then  used  to  estimate  expected  performance  for  all  incumbents  across  all  jobs.  In 
this  process,  job  experience  was  held  constant  (at  4  years)  to  equate  the  estimates  for  people  who 
had  spent  varying  amount  of  time  in  service. 

Three  different  assignment  solutions  were  investigated.  First,  a  baseline  was  established  for 
comparison  purposes  which  set  the  average  performance  of  incumbents  within  each  specialty  to  a 
standard  score  metric  (Mean  =  50;  SD  =  10).  This  reflected  the  efiflcacy  of  the  current  assignment 
system.  Second,  a  linear  programming  algorithm  (Schrage,  1984)  was  used  to  optimize  expected 
performance  across  all  jobs  subject  to  the  constraint  that  all  jobs  be  staffed  with  the  same  number 
of  personnel  as  under  the  present  system.  This  optimal  assignment  represented  the  level  of  overall 
performance  that  might  have  been  achieved  by  capitalizing  on  the  differential  classification 
potential  of  the  ASVAB.  A  third  “random”  solution  was  obtained  by  simulating  assignments 
without  regard  to  aptitude.  The  three  solutions  were  then  compared  on  the  basis  of  the  overall 
average  predicted  performance  across  all  jobs  (Brogden,  1959).  Finally,  the  magnitude  of  gains  in 
predicted  performance  were  expressed  as  a  function  of  the  amount  of  job  experience  needed  to 
achieve  similar  levels  of  performance. 


m.  RESULTS 

An  initial  regression  analysis  of  the  aptitude  and  experience  effects  on  hands-on  performance 
yielded  multiple  Rs  ranging  from  .36  to  .60,  all  significant  at  the  .01  level  (See  Table  2).  Both 
aptitude  and  experience  contributed  uniquely  to  the  predictions.  Results  of  the  assignment 
solutions  (shown  in  Table  3  and  summarized  in  Figure  1)  indicate  an  increase  in  overall  expected 
performance  between  the  random  and  optimized  solution  of  (53.42-48.67)  4.75  units  or 
approximately  one-half  of  a  standard  deviation  unit.  A  comparison  of  the  current  vs.  the  optimal 
solution  showed  a  potential  performance  gain  of  (53.42-49.99)  3.43  units  over  the  current 
baseline. 

The  effects  of  tenure  across  jobs,  held  constant  in  these  comparisons,  were  substantial.  On 
average,  each  one-month  increment  in  experience  resulted  in  a  .23  unit  increase  on  the 
performance  criterion.  Thus,  the  difference  between  the  current  and  optimal  solutions  (3.43  units) 
was  equivalent  to  what  would  have  resulted  if  each  job  incumbent  had  an  additional  14.91 
months  of  technical  experience. 
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Table  2.  Summary  of  Regression  Coefficients  and  Multiple  R’s  by  Specialty  for  Hands-on 
Performance  (N  =  1250) 


Predictor 

. 

SPECIALTY 

JET 

AGE 

RADIO 

PERS 

ATC 

LIFE 

COMM 

GS 

-.028301 

.253341 

-.179182 

-.061659 

.129680 

-.084853 

-.348631 

.106354 

AR 

.242384 

-.007492 

.298839 

.251638 

.362410 

-.010353 

-.008062 

.497242 

WK 

-.127178 

-.441997 

.347863 

.280180 

.022794 

.078684 

.548962 

-.345143 

PC 

-.032372 

.090537 

.246509 

-.090540 

-.298409 

-.166835 

.289150 

.031357 

NO 

.204363 

-.316546 

-187869 

.081718 

-.283656 

-.080212 

.002959 

-.039191 

CS 

.018090 

234308 

-.196472 

.065373 

.343687 

.098737 

-.123094 

.162141 

AS 

.358269 

.261129 

.011111 

.034252 

.129287 

.038467 

.637534 

-.034193 

MK 

-.144909 

.213439 

.114328 

.279252 

-.011882 

.020605 

.624234 

.202154 

MC 

-.058089 

.195572 

.089420 

.019028 

.003228 

.105186 

-.325432 

.340653 

El 

.077108 

.144621 

-.179482 

-.196798 

-.148518 

.124659 

-.161327 

.089314 

EXPERIENCE 

.175352 

.171785 

.254366 

.337048 

.233188 

.276851 

.191850 

.224272 

CONSTANT 

14.937890 

8.654381 

25.825945 

3,954259 

29.522355 

35.448814 

-23.136714 

-18.752804 

MULTIPLE  R 

.362944 

.485988 

.508041 

.508289 

.390429 

.359307 

.598389 

.538034 

NOTE.  ASVAB  subtests  are  abbreviated:  GS  =  General  Science;  AR  =  Arithmetic  Reasoning; 
WK  =  Word  Knowledge;  PC  =  Paragraph  Comprehension;  NO  =  Numerical  Operations;  CS  = 
Coding  Speed;  AS  =  Auto  and  Shop  Information;  MK  =  Math  Knowledge;  MC  =  Mechanical 
Comprehension;  El  =  Electronic  Information.  Specialties  are  abbreviated:  JET  =  Jet  Engine 
Mechanic;  AGE  =  Aerospace  Ground  Equipment  Mechanic;  RADIO  =  Information  Systems 
Radio  Operator;  PERS  =  Personnel  Specialist;  ATC  =  Air  Traffic  Control  Operator;  LIFE  = 
Aircrew  Life  Support  Specialist;  COMM  =  Avionic  Communications  Specialist;  PMEL  = 
Precision  Measurement  Equipment  Lab  Specialist. 


Table  3.  Average  Predicted  Performance  Resulting  From  Three  Assignment  Solutions 


Specialty 

N 

Average  Predicted  Performance 

Random 

Current 

Optimal 

JET 

193 

48.56 

50.00 

51.90 

AGE 

218 

48.63 

50.00 

55.32 

RADIO 

126 

51.74 

50.01 

56.13 

PERS 

176 

49.57 

50-00 

52,83 

ATC 

164 

49.06 

49.88 

52.37 

LIFE 

167 

50.44 

50.00 

50.60 

COMM 

82 

45.20 

50.00 

56.58 

PMEL 

124 

43.92 

49.99 

53.60 

TOTAL 

1250 

48.67 

49.99 

53.42 

NOTE:  Specialties  are  abbreviated:  JET  =  Jet  Engine  Mechanic;  AGE  =  Aerospace  Ground 
Equipment  Mechanic;  RADIO  =  Information  Systems  Radio  Operator;  PERS  =  Personnel 
Specialist;  ATC  =  Air  Traffic  Control  Operator;  LIFE  =  Aircrew  Life  Support  Speci^st; 
COMM  =  Avionic  Communications  Specialist;  PMEL  =  Precision  Measurement  Equipment  Lab 
Specialist. 
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RANDOM  CURRENT  OPTIMAL 

ASSIGNMENT  SOLUTION 

Figure  1.  Average  Predicted  Performance  Resulting  From  Three  Assignment  Solutions 


IV.  IMPLICATIONS 

Evidence  from  the  study  suggests  that  even  though  the  ASVAB  may  be  highly  “g”  loaded 
(Ree  &  Earles,  1990;  Ree  et  al.,  1992,  1994),  it  can  be  configured  to  provide  significant 
differential  validity  in  predicting  hands-on  job  performance  compared  to  the  present  or  random 
assignment  conditions.  The  magnitude  of  the  actual  gains  was  consistent  wth  the  ballpark 
estimates  given  by  Johnson  et  al.  (1990)  in  their  study  using  simulated  rather  than  acmal  data. 
How  much  of  this  potential  is  currently  being  realized?  In  comparing  the  current  assignment 
value  of  49.99  with  the  random  allocation  of 48.67,  the  estimate  is  about  28%  (1.32/4.75  =  .28). 

There  are  probably  upper  limits,  however,  to  how  much  could  be  achieved  since  the 
classification  process  must  operate  within  real-world  constraints.  Applicants  exercise  some 
degree  of  personal  preference  in  accepting  job  offers — and  there  are  other  constraints  (i.e., 
physical  and  medical  qualifications)  that  might  detract  from  an  optim^  classification.  In  sn  era  of 
force  downsizing,  however,  the  services  must  strive  to  maintain  the  highest  levels  of  readiness 
with  a  shrinking  workforce.  More  sophisticated  use  of  aptitude  data  in  the  assignment  process 
would  offer  a  low-cost  alternative  to  more  expensive  interventions  such  as  training  or  job  aids  in 
maintaining  consistently  high  levels  of  performance. 
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